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Executive summary 



The research for this project was conducted from December 1999 through to November 2000. 
This project is about the need for strategies to improve the quality of assessment. Much of the 
research and recommended strategies from this project informed a series of guides developed by 
the Australian National Training Authority (ANTA) to provide assessors with a range of practical 
tools and resources for improving assessment practices. 

In July 2001 the Australian Quality Training Framework focussed on assessment by strengthening 
the requirements of registered training organisations regarding their assessment processes and 
systems. The framework absorbed the National Assessment Principles, recognising that assessment 
must be valid, reliable, fair and flexible. New requirements have been placed on assessors and 
registered training organisations to achieve these principles. 

This research project identified a range of strategies for validating assessment and identified critical 
areas where strategies are essential to increase assessor confidence. An extensive review of the 
literature was conducted, revealing that all research recommended the strengthening of quality 
assurance as a principal strategy for improving the assessment process. 

While there is extensive literature on validity, reliability and moderation in educational testing, 
particularly in the United States, there is limited material on strategies for quality assuring 
assessment in a competency-based environment. The systems in the United Kingdom and New 
Zealand offered a range of options that have some applicability to the Australian VET 
environment. 

The review of the literature reveals that Australian information on possible models and working 
examples is relatively limited. However, eight models have been drawn that have the potential to be 
used by individual or groups of assessors within a registered training organisation to support 
improved assessment practice and decision-making (see p.28). 

In addition to this report, a resource for assessors and training provider managers was also 
developed, called Maximising confidence in assessment decision-making: ^source kit for assessors. This is 
available at <www.ncver.edu.au/research/proj/nr9029.pdf>. 
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Preface 



There have been significant developments in the vocational education and training (VET) sector in 
the areas of quality assurance and consistency of assessment since the completion of this research. 

The research for this project was conducted from December 1999 through to November 2000. 
This project and a number of other recent national research projects with a focus on assessment 
have influenced policy-makers in the sector as they have highlighted the concern felt by assessors, 
VET managers and researchers about the need for strategies to improve the quality of assessment. 

In July 2001 the Australian Quality Training Framework focussed the VET sector’s attention on 
assessment by strengthening the requirements of registered training organisations regarding their 
assessment processes and systems. The framework has absorbed the National Assessment 
Principles, recognising that assessment must be valid, reliable, fair and flexible. There are now new 
requirements placed on assessors and registered training organisations to achieve these principles. 

A number of sections in the framework deal specifically with assessment and include the registered 
training organisations’ requirement to develop assessment strategies ‘in consultation with 
enterprises/industry’ (ANTA 2001) and to validate assessment strategies. Section 9.2 states: 

The RTO [registered training organisation] must validate its assessment strategies by: 

(i) Reviewing, comparing and evaluating the assessment process, tools and evidence 

contributing to judgements made by a range of assessors against the same competency 
standards**, at least annually; and 

(11) Documenting any action taken to improve the quality and consistency of assessment. 

Note: ** These may be internal processes with stakeholder involvement or external validations with other providers and/or 

stakeholders 

Assessors consulted in this research agreed that assessment validation was the most viable means 
of achieving quality, consistency and improving the confidence of assessors. Validation was also 
seen as a major hurdle for the majority of registered training organisations. 

The need to comply with Australian Quality Training Framework standards moves assessment 
validation from ‘good to do’ to an essential requirement for training providers. Perhaps this shift to 
compliance will have an impact on improving assessors’ confidence. It will be interesting to see 
whether assessors report that their confidence in assessment decision-making has changed over the 
next two years as the framework is implemented. 



Resources to assist assessors 

The Australian National Training Authority (ANTA) has managed several initiatives to improve the 
quality of assessment and facilitate the implementation of training packages. One of these projects, 
the Training Package Materials Project, involved the development of a series of ten guides to 
provide assessors and managers of assessment processes within the VET sector with a range of 
practical tools and resources for improving assessment practices. Members of the research team 
from the Maximising Confidence in Assessment Project collaborated on a number of the guides in 
the ANTA products. Much of the research and recommended strategies from this project have 
informed the development of the guides. 
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Introduction 



Rationale for the project 

The purpose of this research project was to investigate the critical factors that impact on the 
confidence of assessors in making assessment judgements. The project grew from concern being 
voiced by assessors and registered training organisations about consistency in assessment decisions. 
For mutual recognition of qualifications to work successfully between registered training 
organisations, there needs to be widespread confidence in the consistency of assessment decisions. 

Assessment against competency standards involves collecting evidence and then forming a 
judgement about whether or not competency has been achieved. Several factors impact on these 
judgements, including the skill and experience of assessors and the clarity of the particular 
competency standards themselves, and these can influence the assessment decisions made. 

This research project identified a range of strategies for validating assessment currently being used 
by assessors in training providers, and identified critical areas where strategies are essential to 
increase assessor confidence. In addition to this report on the research process and findings, a 
resource kit for assessors and managers in registered training organisations was also developed as 
part of the project. 

The resource kit includes a set of recommended strategies for registered training organisations to 
assist them with the quality assurance aspects of assessment. This was developed from material 
collected as part of the research project and based on recommendations made by practitioners 
consulted during the research. 



Definition of terms 

The definitions for quality assurance, verification process and validation process provided in the 
Training Package for Assessment and Workplace Training (ANTA 1998) were used in the project 
issues paper and at workshops during the project consultation process. Participants at the 
workshops agreed that these definitions should be the ones adopted for this project. Definitions of 
terms used in the project are provided in the literature review. 



Terminology 

With the introduction of the Australian Qualifications Training Framework, there has been a 
change in the terminology used to describe some assessment quality assurance processes. 
‘Assessment validation’ has been adopted as the term describing the processes a registered training 
organisation needs to put in place to ensure the quality and consistency of its assessment. While 
undertaking this project, the team used definitions for assessment validation and assessment 
verification from the Training Package for Assessment and Workplace Training (ANTA 1998). 
However, these definitions have now been subsumed under the one term, validation, and these 
changes in terminology can be seen in the report and project resources. A set of materials has been 
developed from this research as a kit to assist assessors. 



NCVER 



7 




While there is no absolute definition of the term ‘confidence’, for this project it represents a sense 
of trust in the quality of the information provided to learners, the validity of assessment tools, and 
the reliability and fairness of assessment procedures. It also involves the sufficiency of evidence, 
correctness of the interpretation, and the accuracy of the recording and reporting of assessment 
results. 



Research questions 

The research questions addressed by the project are: 

What factors contribute to the development of confidence in assessment decision-making? 

What strategies are assessors using to ensure that their judgements of competence are valid, fair 
and reliable, and based on the collection of sufficient evidence? 

What strategies are registered training organisations using to support the verification and 
validation of assessment? 
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Literature review 



Introduction 

Until 2001, the national principles, standards and operating protocols that comprised the Australian 
Recognition Framework were seen as sufficient to maintain the quality of assessment in registered 
training organisations delivering vocational education and training. These measures included: 

national core standards for registration 

national product/ service standards governing the capacity to conduct assessment which met 
requirements of training packages 

national operational protocol for an external review process involving the validation of the 
registered training organisations’ internal monitoring/ review arrangements 

national training packages consisting of assessment guidelines which included an assessment 
system overview, assessor qualifications and training, guidelines for designing assessment 
resources, guidelines for conducting assessment, and sources of information on assessment 

professional development 

national Training Package for Assessment and Workplace Training that also outlined the 
responsibilities of training providers in the quality assurance of assessment. 

All these measures were designed to support quality assessment and maintain consistent training 
outcomes. Consistency specifically relates to the comparability of results for learners who are 
assessed in a range of contexts by a variety of assessors against the same competencies. The 
emphasis on consistency is deemed to be critical as it forms the basis for the mutual recognition of 
VET qualifications. 



VET assessment under review 

The credibility of the VET system, however, was brought into question by a series of reviews at 
both national and state levels (Schofield 1999a, 1999b; Smith 2000; ANTA 2000a, 2000b). These 
studies brought to the forefront a range of concerns about the general quality of assessment across 
the VET sector. In particular, it was noted that the National Assessment Principles, standards and 
protocols provided under the Australian Recognition Framework were insufficient to ensure the 
quality or consistency of either the assessment processes or the judgements being made by 
assessors. 

In their 2000 submission to the Senate Inquiry into the Quality of Vocational Education and 
Training in Australia, Business Skills Victoria suggested: 

Anecdotal evidence is increasingly being provided to BSV [Business Skills Victoria] that many 
enterprises are concerned about the reliability and validity of assessments performed by RTOs 
[registered training organisations] in the workplace. Some of these concerns are caused by a 
lack of familiarity with assessment as a model which can lead to credentiaUing but also the 
concerns are caused by assessment processes which are ‘tick box’ and with apparently 
insufficient mechanisms to ensure adequate evidence has been collected. Moderation and/ or 
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validation of a percentage of assessments by an external expert would build the credibility and 
quality of the assessment process. (Business Skills Victoria 2000) 

This was but one of many submissions to the Senate Inquiry which queried the quality of existing 
assessment practice and outcomes. These submissions, when combined with the findings of the 
investigations conducted by Schofield, Smith and others, confirmed that there were serious 
limitations in the audit process that formed the basis for quality assurance in the national system. It 
also indicated significant deficits in the expertise of many VET assessors. 

By 2000, concerns about the quality of assessment saw the generation of a strategic evaluation by 
ANTA to look at consistency across all VET jurisdictions. This review again found broad-ranging 
concerns about the nature and quality of assessment being conducted in the VET sector. Some 
major factors were identified as influencing the consistency of VET assessment. These included the 
ineffectiveness of audit to improve assessment, concerns about the quality and extent of training 
being undertaken by VET assessors and the level of their technical expertise, and the lack of 
rigorous quality assurance within registered training organisations. 



Limitations of the external audit process 

Reviews of the system revealed that the quality assurance arrangements in the Australian 
Recognition Framework were very much placed at the front-end of the system. This was due to the 
focus on registration of training providers, the way that assessors were initaUy trained, and the 
provision of standardised assessment information via the assessment guidelines within training 
packages. As Schofield (1999b) identified, there was no systematic approach to checking the 
outcomes of training to determine the achievement of the designated standards (p.52). 

Smith (2000) also stated his misgivings about the effectiveness of the audit process as a quality 
assurance mechanism. He suggested that with such an over-emphasis on record-keeping and other 
physical evidence — rather than a more rigorous evaluation of assessment procedures, practices and 
judgements — the desired outcome of a quality assured assessment system under the Australian 
Recognition Framework was not being achieved. Those interviewed for Smith’s study highlighted 
the need for a process of review or moderation to enhance the consistency of assessment 
approaches, the processes of assessment and the final judgements about learner competence. 

While audit was not seen to be an effective way of improving the situation, a strategy was devised 
that would help in generating consistent assessment practice. This involved checking assessment 
instruments, processes and the ultimate decisions being made by assessors. 

The belief is that a moderation system would not only set, propagate and check assessment 
standards but would also facilitate the sharing of ‘good practice’ approaches to assessment 
across the system. (Smith 2000, p.l6) 

Tower, Bloch and Harvey (1999), however, had already offered a warning about placing too much 
faith in regulatory processes and standardised information and procedures to achieve greater 
national consistency. They suggested that highly skilled assessors, who could undertake their tasks 
with confidence, were essential to the achievement of more positive training outcomes (p.2) 



Deficits in assessor expertise 

The level of expertise required to carry out assessment in vocational education and training has 
been well acknowledged. VET assessment demands a substantial amount of tacit knowledge and 
judgement on the part of assessors and a considerable degree of responsibility is entailed in making 
judgements about learner performance (Jones 1999; Docking 1997). Poor judgements about learner 
competence ultimately have significant consequences for the credibility of the VET system 
(Fechner & Hill 1997). 



10 



Maximising confidence in assessment decision-making 




As Docking suggested: 

The implications of getting the judgement incorrect, however, are also likely to be significant. 

The dangers of incompetent assessment go far beyond the classroom and can impact on a 
person’s whole life and can harm their future clients or employers. (Docking 1997, p.l9) 

Given the findings of the reviews, there was extensive evidence to indicate that deficits existed in 
assessor expertise. Many of these were the result of ineffective initial training, a lack of ongoing 
support or professional development, and neglecting whether assessors’ technical skills were up to 
date. 

In conducting a review of the competency standards for assessment and workplace trainers, GiUis, 
Griffin, Trembath and Ling (1998) noted the lack of rigour and quality assurance in the training of 
workplace trainers and assessors as major concerns. They commented that such training ‘was 
demonstrated to be inadequate in a significant number of settings’ (p.l85). In his Queensland 
study, Smith confirmed this view of pre-service training programs, and commented on what he 
called a ‘pervasive and deep concern’ in relation to assessor expertise (p.lO). 

It was reported that VET assessors training frequently is conducted by people who have no 
expertise in assessment beyond the level of the course they are conducting, and so lack the 
capacity to significantly lever quality assessment into the system. (Smith 2000, p.ll) 

Consultants reporting on outcomes of the scoping study for the national assessment initiative 
recommended to the National Quality Training Council that a range of issues be considered in the 
review of the Training Package for Assessment and Workplace Training. Most focussed on the 
skills and qualifications required by assessors in light of the introduction of training packages. One 
critical suggestion was: 

. . . the inclusion of units of competency within the Training Package for Assessment and 
Workplace Training (BSZ98) that address the skill requirements for assessors to engage in 
Assessment Review or moderation/ verification practices. (ANTA 2000a, p.38) 

In addition to concerns registered about initial assessor training, the dearth of ongoing support for 
assessors was consistently raised as a major issue in a large number of studies (Docking 1997; 

GiUis, Griffin, Trembath & Ling 1998; Dickson & Bloch 1999; Tower, Bloch & Harvey 1999). In 
her recommendation to the Tasmanian Accreditation and Recognition Committee, Schofield 
acknowledged the problem and proposed the development of an ‘assessors’ forum’ to maintain 
their skills. It was indicated that: 

This Forum should explicitly foster greater professionalism and ethical practice in workplace 
assessment and a climate conducive to self-regulation by encouraging assessors to: 

V share assessment strategies and tools 

V participate in voluntary assessment moderation activities; and 

V consider and create best practice examples of workplace assessment. 

(Schofield 1999a, p.xU) 

The strategic evaluation of consistency in assessment confirmed that there was a need to enhance 
both the initial and ongoing training of assessors. It was also suggested that assessors needed better 
advice on the gathering and evaluation of evidence and quality exemplar assessment materials to 
support better and more confident decision-making (ANTA 2000b). 

It was apparent from these studies that inconsistencies in assessment practice, limitations in 
assessor training and the lack of ongoing professional development were having an influence upon 
assessment outcomes. When combined with ineffective quality assurance processes, these issues 
were Ukely to have a detrimental effect on the confidence of aU stakeholders involved in vocational 
education and training. 
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Quality assurance of assessment 

The Training Package for Assessment and Workplace Training states that quality assurance 
mechanisms are an integral part of any system (p.l36). Quality assurance is defined as: 

... a planned and systematic process of ensuring that the requirements of the assessment 

system, competency standards and any other criteria are applied in a consistent manner. 

(ANTA 1998, p.l36) 

Consistency is absolutely central to this concept of quality assurance, because it is recognised that 
assessment judgements are made on the basis of assessor experience or tacit knowledge of their 
technical domain (Jones 1999). Assessors are using internalised models of competence to 
determine the quality of learner performance, and not all assessors perceive competence in a similar 
way (J31och, Clayton & Favero 1995). Given these circumstances, it is crucial that a system of 
checks and balances be put into place to protect both the assessors and key stakeholders in the 
process. 

A framework for a comprehensive quality assurance strategy for a VET assessment system was 
established by Toop, Gibb and Worsnop (1994). The mechanisms they proposed included the 
screening and training of assessors, verification of assessment decisions, appeals mechanisms and 
processes, and a review of the assessment system. These strategies were extended to some degree 
by Alexander, who listed four criteria for quality assurance. These included the use of qualified 
assessors; an established validation, feedback, appeals and verification process; industry-endorsed 
assessment guidelines; and the provision of industry audit of the assessment process (cited in 
Docking 1997, p.ll). 

GiUis, Griffin, Trembath & Ling (1998) suggested that at the system level, quality assurance 
mechanisms can and should be integrated in a number of ways. They proposed a range of strategies 
to improve the consistency of VET assessment. These included closer adherence to assessment 
guidelines and greater use of resources contained in the nationally endorsed training packages, and 
monitoring and auditing training organisations through the registration process. Other strategies 
involved quality assurance processes at the assessment system level (inside training providers), 
quality assurance processes at the assessment experience level, and enhanced initial training of 
assessors followed by ongoing support and development (p.2. 12-2.28). 

Each of the frameworks entailed elements of both quality assurance and quality control. As 
Maxwell (2001) suggested, the ‘feed-forward’ approach of quality assurance sets in place clearly 
defined procedures designed to generate the desired outcomes, while the ‘feed-back’ or quality 
control components are more designed to measure whether the outcomes are themselves 
acceptable (p.3). 



Strategies for quality assuring assessment 

As described by GiUis, Griffin, Trembath and Ling (1998), the quality assurance processes at 
assessment system level and assessment experience level include moderation, verification and 
validation. 



Moderation 

Interestingly, whUe the Training Package for Assessment and Workplace Training regularly 
mentions the term ‘moderation’, no definition of the term is included in the glossary. Maxwell 
defines the term as: 

... a particular process of quality control involving the monitoring and approval of assessment 
procedures and judgements to ensure there is consistency in the interpretation and application 
of the performance standards. (Maxwell 2001, p.3) 
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Thus, by moderating assessments, it is possible to identify the inaccurate application of standards 
or other inconsistencies in approach across a number of assessors (Foyster 1995). 

Approaches to moderation include statistical, visitation, and group or consensus moderation. The 
latter approach involves groups of assessors meeting to discuss assessment procedures, processes 
and decisions in order to identify anomalies or confirm the consistent application of standards 
across a range of assessments made by the group. This generally necessitates a process of sampling 
which may include: 

selecting a sample of each candidate’s performance for further consideration 
selecting all the work of a sample of candidates 
a combination of the above. 

For Foyster, consensus moderation in particular has the added advantage of building the 
confidence of assessors because it provides an opportunity for professional development and a 
more consistent understanding of the processes and benchmarks required. The advantages of 
shared understanding of standards, evidence requirements and sufficiency of evidence were also 
highlighted in the ANTA strategic evaluation of consistency of VET assessment (ANTA 2000b). 

Flowever, some warnings about moderation are evident in the literature. In reporting on an English 
research project looking at moderation, Radnor and Shaw commented: 

. . . these processes of moderation are problematic not only in technical and logistic terms, but 
also because they raise issues of power-sharing . . . They also raise issues of dignity, and 
professional self-respect, of involvement in decision-making, of skill and training, of the 
confidence teachers have in their own judgement, and of the lack of agreed and tested 
principles and working models, as opposed to generalEed, well-intentioned official 
statements. (Radnor & Shaw 1995, p.27) 

Further, Foyster (1995) suggests that unless there is a formal system in place used by all in the same 
manner, there is no point attempting moderation at all. Moderation is a process that involves the 
measurement of assessors’ judgements against a standard and, as such, is dependent on social 
interaction between assessors. Thus, a formal process needs to be set in place in order to resolve 
differences of opinion that may arise (p.7). 

Verification 

The Training Package for Assessment and Workplace Training defines the process of verification 
as ‘the means of ensuring that the assessment decision is consistent and reliable. The process may 
involve having another assessor(s) confirm the assessment decisions’(ANTA 1998, p.l37). 

In the United Kingdom, the term verification is defined by the Qualifications and Curriculum 
Authority as: 

. . . one aspect of quality assurance which relates to the day-to-day delivery of the NVQs 
[National Vocational Qualifications], rather than the quality assurance of the system as a 
whole. It is the process of monitoring assessment practice to ensure that assessment decisions 
are consistently accurate. (Qualifications and Curriculum Authority 1999, p.2) 

Gilks, Griffin, Trembath and Ling (1998) present a much more complex definition for the process. 
For them, verification entails the adjustment of assessment decisions against external criteria, with 
the goal of achieving consistent interpretation and application of standards. Furthermore, they 
contend that a verification process achieves uniformity when those involved work towards a 
common understanding and usage of concepts, terminology and application. This definition fits 
quite comfortably with that of ‘moderation’. 

Toop, Gibb and Worsnop (1994) suggest that ‘verification’ is the term used to describe the checks 
and balances in the assessment system. Its primary purpose is quality assurance and it has the 
potential to enable users to have confidence in the system and the certification that results from it. 
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Verification may occur before, during, and after an assessment, while moderation is the main 
verification process which happens after the assessment is made. The procedures which Toop, 
Gibb and Worsnop suggest can be included in each of these stages are set out in table 1 . 



Table 1: Timing and procedures involved in verification 



Timing 


Procedures 


Before assessment 


Assessment centres having to demonstrate they have the required facilities and 
equipment 

Assessors meeting established criteria to qualify as assessors including expertise in 
occupational area, expertise in process of assessment and availability to assess 

Reviewing written materials which are given to candidates 
Conducting trials of assessment materials before their widespread use 


During assessment 


Using more than one assessor 

Assessors exchanging a selection of evidence and reassessing other materials 
independently 

Sampling by internal or local verifiers 
Sampling by external verifiers 


After assessment 


Statistical monitoring of awards 
Moderating instruments 



Validation 

The Training Package for Assessment and Workplace Training defines validation as: 

... a process to ensure assessment tools, procedures and decision criteria lead to a correct 
decision when used to assess competency. The process may involve having both technical and 
assessment specialists review the assessment tools and procedures for validity. 

(ANTA 1998, p.l37) 

Until the revision of the Australian Recognition Framework, Validation’ tended to be used 
interchangeably with Verification’ and, whilst the training package defined the process, it provided 
no guidance on the activities that validation might have entailed. 

Black (1 993) supports the concepts inherent in the above definition and suggests that the scrutiny 
of assessment tools is a particularly important aspect of any quality assurance process, especially if 
it is a participatory activity among a range of assessors. However, he believes that the principle of 
evaluating assessment tools as a matter of course is even more vital than the review process. 

The language of quality assurance 

The definitions provided for moderation, verification and validation reveal that there are 
considerable contradictions and confusion in the language associated with quality assurance of 
competency-based assessment. Interchangeable terminology and definitions with multiple 
meanings do litde to support consistency in processes, procedures and decision-making. Such 
terminology requires clarification and consistent usage if practitioners and policy-makers are to 
establish mutual understandings about assessment. 



Quality assurance models 

Quality assurance strategies are in place in both the United Kingdom and New Zealand VET 
systems. The mechanisms employed in the monitoring of training outcomes in both countries 
include both the ‘feed-forward’ and ‘feed-back’ components described by Maxwell (2001). 
However, the extent and rigour in the review process differ considerably between the two 
countries. 
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The National Vocational Qualifications model 

In the United Kingdom, the Qualifications and Curriculum Authority contends that effective 
quality assurance is the critical element in building consistency and confidence in National 
Vocational Qualifications (National Council for Vocational Qualifications 1997a, 1997b). As a 
consequence of this emphasis on quality assurance, a highly regulated approach to assessment in 
National Vocational Qualifications has been developed. 

Within the National Vocational Qualifications system, a dual layer of monitoring makes up the 
quality assurance strategy, and entails internal and external verification. AtvardingHodies’ 

Common Accord 1997 provides the guidelines for quality arrangements in the system, while 
Implementing the National Standards for Verification sets out clearly how the national standards for 
assessment and verification units are to be assessed and verified. 

All training centres delivering National Vocational Qualifications are required to undergo 
registration and, as part of this process, must be able to demonstrate their capacity to undertake 
assessment. They are also required to implement rigorous moderation procedures to monitor and 
evaluate all aspects of their own assessment. The specific elements of this process of internal 
verification include: 

V monitoring the conduct of assessment 

V sampling of candidate evidence to verify assessment decisions 

V assuring the quality of the systems and procedures used for assessment and verification 

V provision of support and advice for assessors 

V maintenance of assessment and verification records to allow analysis by the centres delivering 
the training and the awarding body. 

(National Council for Vocational Qualifications 1997b) 

According to the Qualifications and Curriculum Authority requirements, individuals nominated to 
undertake the role of internal verifier within a centre are required to ensure that assessors and their 
assessment practice meet both the awarding body’s criteria and the national standards for 
assessment. They are also expected to provide feedback to assessors in the centre, and ensure that 
verification and assessment records are completed and deal with disputes (National Council for 
Vocational Qualifications 1997b, p.lO) 

External verifiers determine whether the quality of both assessment and verification meets national 
standards. The external verification process begins when external verifiers meet with internal 
verifiers, assessors and candidates to help establish the key points in the assessment and recording 
process. External verifiers undertake sampling and check the record-keeping system with the 
centres. 

The system in the United Kingdom is centralised and highly regulated. Studies by Black (1993), 
Eraut (1994), Lester (1996, 1997, 1999) and Konrad (1999) highlight some of the critical issues and 
concerns that they have with the approach adopted for assessment of National Vocational 
Qualifications. In particular, they focus on the quality control nature of the United Kingdom system 
which includes increasing external monitoring and standardisation. Lester (1996) believes that the 
solution to consistent high quality assessment outcomes is not one of quality control, but rather 
quality assurance. He suggests that the solution primarily rests with the ongoing professional 
development of the people conducting the assessment of National Vocational Qualifications. 

Lester considers that increasing the quality control measures is bound to ultimately have 
detrimental outcomes. 

The result is likely to be that while public confidence is increased, validity suffers as 
assessment increasingly concentrates on factors which are amenable to checks and controls. 

(Lester 1996, p.3) 
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More specifically, Konrad (1999) suggests that the complex nature of the role of internal verifier is 
often under-resourced and those carrying out this task have barely adequate initial education and 
training to undertake the activities effectively. 

Confirming this view, Eraut (1 994) comments that regulation alone will not achieve the desired 
outcomes of a quality assured assessment system and that a greater focus on building the skills and 
knowledge of assessors is likely to have greater impact. He offers the following comment: 

Evidence suggests that once established by training and regular communication, a community 
of assessors is able to ensure sufficiently standard use of criteria; but that it is easy for 
standardization to slip if training and communication are not regularly maintained. The 
training of assessors and verifiers is another essential component of quality assurance, because 
assessment and verification are themselves professional processes requiring special expertise. 

(Eraut 1994, p.207) 



The New Zealand model 

The New Zealand Qualifications Authority adopts a somewhat less rigorous approach in 
implementing the New Zealand Qualifications Framework, and its quality assurance strategy is 
underpinned by the Principles of Best Practice Moderation. These principles assert that best 
practice moderation occurs when it is based on partnerships between assessors and other assessors, 
and also assessors and moderators. In addition, it needs to be ongoing, as well as an evolving and 
educative process for assessors and moderators (New Zealand Qualifications Authority 2001). 

Unit standards are established by standards-setting bodies, providers are registered and assessment 
materials are offered in some industry sectors by the relevant industry training organisation. There 
are also unit standards for assessment, and it must be conducted by those who have completed the 
appropriate training. 

The main quality assurance mechanisms employed concentrate on moderation and audit of 
registered private providers and approved government training establishments. Thus, much of the 
emphasis is placed on monitoring the assessment processes and judgements after assessment has 
been completed. 

Industry is actively involved in the moderation process. Assessments conducted against unit 
standards that are drawn from specific industry sectors are moderated through processes 
established by the relevant industry training organisations. 

Importantly, the New Zealand Qualifications Authority requires providers of training to develop 
their own quality assurance systems. However, aU unit standards set requirements for moderation, 
and training providers and standard-setting bodies are responsible for ensuring that assessors 
participate in the designated process for ensuring validity and consistency in assessment. 

The process of review is set out in moderation action plans, which provide the framework for 
evaluating aU aspects of assessment. These plans include information on the key roles of people 
involved in the moderation process, the unit standards to be moderated in a particular year, and the 
funding of the moderation process. Other information involves the proportion of candidate 
assessments to be moderated in a particular year, the frequency of moderation, moderation record- 
keeping requirements, and the approaches for dealing with any non-compliance with moderation 
procedures (http:/ /www.nzqa.govt.nz/). 

Most moderation procedures established by the industry training organisations include provision 
for both internal, interprovider, and national moderation. Internal moderation, like internal 
verification in the National Vocational Qualifications system in the United Kingdom, is focussed 
on achieving consistency between assessors judging against the same unit standard in an 
organisation. Consensus or group moderation is the major mechanism employed to ascertain the 
accuracy and comparability of assessment decisions across a range of training organisations. 
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Interprovider moderation involves a training provider linking with another provider offering 
similar vocational courses and levels of training to conduct a review of assessment processes and 
evaluate judgements on a sample of student work (New Zealand Qualifications Authority 2001). 
Additionally, national networking of subject moderators and evaluation of assessment systems at 
the time of accreditation and re-accreditation is designed to assure the quality, consistency and 
credibility of assessment of national qualifications. 

The approach adopted by New Zealand, however, is one that is highly dependent on resources 
being made available to support the system of monitoring and evaluation. Thus, Maxwell suggests, 
opportunities for assessors to network to develop real consensus are actually quite limited 
(2001, p.26). 



Conclusion 

A number of research projects and reviews of the delivery of vocational education and training in 
Australia revealed major problems with the conduct of assessment in the sector. Despite the 
National Assessment Principles, the standards and the operating protocols provided by the 
Australian Recognition Framework, consistency and quality were recognised as problematic. With 
limited quality assurance and questionable levels of assessor expertise, questions were raised about 
the value of the qualifications generated within the VET system. The confidence of key 
stakeholders, including assessors, was also seen to be quite low. 

All research and reviews recommended the strengthening of quality assurance as the principal 
strategy for improving the quality and consistency of assessment processes, assessment tools and 
assessment decision-making. The systems in both the United Kingdom and New Zealand offer a 
range of options that have some application to the Australian VET environment. 

It is apparent, however, that the implementation of a broad-ranging quality assurance and quality 
control regime will not in itself generate the improvements required. As Smith suggests: 

. . . the achievement of assessment quality requires an integrated package of initiatives: relevant 
and practice-oriented pre-service and professional development training; a combination of 
both progressive (‘formative’) and summative assessment; a genuine ‘team’ effort between 
professional trainers and appropriate industry/ employer representatives; on-going research 
into, and dissemination of, best practice; clearly defined standards for both initial and on- 
going ‘registration’ as a trainer (as distinct from a training organisation); and an effective 
system of moderation and audit. Unless all of these components are addressed in an 
integrated fashion, quality of assessment is unlikely to be achieved. (Smith 2000, p.29) 
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Methodology 



Research methods used 

The methodology for this research project was designed as an iterative process with models and 
strategies for the validation and verification of assessment being presented to practitioners during a 
series of workshops. An issues paper was prepared for use in the project workshops. This paper is 
included as appendix 1 , and highlights relevant sections of the literature review and the areas of 
concern identified by an initial project focus group. The project workshops were designed to 
collect data on existing quality assurance strategies within registered training organisations, and 
identify potential strategies and models which could be adopted by other training providers to 
increase confidence in assessment decision-making. 

Initially, possible models and strategies to increase confidence in assessment were sourced from the 
review of the literature. These were then refined through several stages of input from workshop 
participants resulting in a set of recommended strategies, guidelines and scenarios for registered 
training organisations designed to assist with quality assurance of assessment systems and 
processes. 

Six project workshops were held to gather data from practitioners about the range of strategies that 
training providers had in place to assure the quality of their assessment systems. This process 
consisted of two series of workshops with three workshops in each series. The workshops were 
designed using a participative model so that assessors could establish common ground, search for 
workable strategies and develop creative solutions. 

Following the data collection and strategy development process that occurred in both series of 
workshops, the strategies developed were reviewed and refined as resource materials for assessors 
and registered training organisations. This resource material included strategies identified as 
appropriate to assure quality in assessment, models and case studies, for implementation by 
assessors and training providers. These resources were distributed to a range of assessors and 
organisation representatives for comment, and final revisions were then made. 
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Figure 1 : Overview of project methodology 




Summary of focus groups and workshops 

Initial project focus group 

An initial project focus group was held in Sydney in November 1999. Participants raised issues they 
believed to be impacting on the quality of assessment and the confidence of assessors and others in 
the VET sector. These issues can be grouped as: 

the lack of consistency in assessment decisions and practice 

the new demands placed on assessors and assessment with the implementation of training 
packages 

the lack of rigorous quality assurance processes 

concerns about the quality of assessor training programs and ongoing support for assessors. 

These identified issues were used as a framework for the collection of material from registered 
training organisations and practitioners during the project. 

The project literature review was completed. This review provided information about the range of 
existing registered training organisation strategies relating to quality assurance of assessment both 
in Australia and internationally. Strategies identified in the literature review were used in the 
development of project material. 

Workshops 

The format of the two series of workshops was similar and consisted of two major areas of focus. 
Firstly, workshop participants provided feedback on how their organisations addressed a range of 
areas impacting on the quality of assessment. Secondly, participants provided feedback on a 
collection of strategies put together by the project team as possible quality assurance strategies that 
training providers could adopt to increase assessor confidence. 

Selection of workshop participants 

Registered training organisations in the selected industry areas of study (horticulture, information 
technology and children’s services) were targetted to nominate representatives. Representatives 
from these organisations, who regularly assess or supervise the assessment process and would be 
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able to provide an overview of their organisation’s assessment policies and approach, were asked to 
participate in project workshops. 

State industry training advisory bodies and state training authorities provided details of suitable 
registered training organisation representatives and assessors from a range of states to invite to the 
workshops. Workshop participants thus included representatives from private and public sector 
training providers, group training companies and industry training advisory bodies. Participants 
also had experience assessing in a range of different contexts including enterprises and VET in 
Schools. They included a number of training provider representatives whose responsibility included 
the establishment and maintenance of registered training organisation quality systems with a focus 
on assessment. A list of project participants is attached as appendix 2. 

Series 1 workshops 

Three industry-specific workshops were held in Canberra and Melbourne to gather data from 
participants on existing strategies for quality assurance of assessment in registered training 
organisations. Workshop participants were drawn from New South Wales, the Australian Capital 
Territory, Victoria, Tasmania and South Australia. The following range of organisations was 
represented in the first series of workshops: 



Table 2: Series 1 workshop participants 



Organisations 


Number of participants 


ITAB 


3 


Public sector RTO 


16 


Private sector RTO 


3 


VET in Schools provider 


2 


Group training organisation 


1 


State training authority 


3 


Enterprise RTO 


1 



Notes: ITAB = industry training advisory body 

RTO = registered training organisation 

Series 2 workshops 

Unlike the first series, the second series of workshops held in Brisbane and Sydney were not 
industry specific. Each workshop group included a range of participants from the three project 
industry areas to encourage debate on the wider applicability of the models to a range of industry 
areas. The following organisations were represented: 



Table 3: Series 2 workshop participants 



Organisations 


Number of participants 


Public sector RTO 


11 


Private sector RTO 


13 


Group training companies 


1 


VET in Schools sector 


3 


State ITAB 


1 



Notes: RTO = registered training organisation 

ITAB = industry training advisory body 

Participants at both series of workshops were provided with the project issues paper as background 
reading and preparation for the workshops. They were also provided with a list of areas of concern 
that had been identified by the initial focus group as critical for the registered training organisation 
in the maintenance of assessor confidence. These areas were: 
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registered training organisation assessment policies 
assessment information provided to learners 
a documented oudine of the assessment process 
guidelines to assist in the development of assessment tools 
processes or documentation to support assessors 
procedures for the review of assessment. 

Prior to attending the workshop, participants were asked to gather information about how these 
areas were handled by their organisation and, where possible, provide examples of existing 
strategies for discussion. Participants were also asked to provide any other strategies used to help 
ensure the quality of assessment. 

There was consensus among participants at each of the workshops that this list of areas of concern 
was an accurate representation of the factors impacting on assessor confidence. This confirmed 
that the initial focus group had been a valuable process in identifying the issues. 



Review of methodology 

The methodology adopted provided a good opportunity to work with groups of assessors, gather 
input about their organisation’s strategies, and gain feedback about how useful specific strategies 
would be in improving assessor confidence in their contexts. For these groups to function 
effectively and allow for input from all participants, numbers needed to be restricted to about 1 5 
per group. At times this created a difficulty in identifying participants from a sufficiendy diverse 
range of organisations and regions. Workshop participants came from different types of registered 
training organisations and from six different states. However, the sample numbers in each group 
were so small that it is difficult to say whether they represented the same views as the full range of 
training providers working in the Australian VET context. 

In addition to the participants who attended workshops, the project team interviewed and spoke to 
over 40 other registered training organisation representatives and assessors, many of whom were 
referred by workshop members as contributing a significant strategy or process to improve 
assessor confidence. Some of the additional assessors interviewed provided material for the case 
studies in Maximising confidence in assessment decision-making;, ^source kit fior assessors (Booth et al. 2002). 

Identifying the most appropriate individual in each organisation to attend workshops was 
sometimes difficult. Full-time assessors who attended had wide knowledge about assessment 
practices in their registered training organisation, but sometimes did not have a good understanding 
of areas that related to the assessment system and how it functioned. 

Where the workshop participant had a close understanding of their organisation’s assessment 
system, they may not have been familiar with the actual assessment practices. To some extent, 
these gaps in knowledge were counteracted by giving all workshop participants background reading 
to prepare them for the tasks they would be undertaking in workshops. They were also provided 
with a series of areas to research prior to the workshop, so they were able to contribute details of 
their registered training organisation’s practice in the listed areas. 



Limitations of the study 

At the time this research was conducted, there were differences across states in how state training 
authorities were conducting compliance audits for training provider registration. This influenced 
the types of strategies for assessment validation that registered training organisations in those states 
had established. It would have been beneficial to have included larger numbers of assessors from 
each state to be able to determine whether there was significant difference in assessment validation 
strategies in certain areas or industries. 
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The three industry areas selected for the project were initially chosen because they each had 
assessors working in a wide range of contexts and geographic locations. It is difficult to generalise 
as to whether the strategies identified for increasing assessor confidence in these three industries 
are indicative of what is happening in all industry areas in the VET sector. Many workshop 
participants were assessed in areas other than the one they were representing at that time, and other 
participants were familiar with assessment practices in other industry areas in their organisation. 
Thus, there was incidental input about practices in other industry areas. 

Despite these limitations, the study has provided an opportunity to develop thinking about how to 
assure quality in assessment, particularly with the range of practitioners working in such a variety of 
assessment contexts. 
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Factors which contribute to 
the level of confidence in 
assessment decision-making 



Existing strategies 

Existing registered training organisation strategies relating to assessment which impact on assessor 
confidence and the overall quality of the assessment process are summarised below. This summary 
was compiled from feedback provided by participants at both series of workshops. 

1. Assessment policy 

Workshop participants all identified that their registered training organisation had some form of 
assessment policy, but these policies ranged considerably in content. In addition, there was 
widespread confusion about what actually constitutes policy. Often there was overlap between 
assessment policy, procedures and the information provided to candidates. Workshop participants 
observed that the majority of current registered training organisation assessment policies focus on 
grievance and appeals processes, but participants felt that assessment policies should contain a 
code of practice for assessors. There was agreement that an assessment policy should be readily 
accessible to all assessors in the organisation. 

In smaller training providers, there was usually no-one qualified to write policy and very litde 
guidance provided on what should be included. There was widespread agreement about the need to 
continually review and update assessment policy so that it remains relevant to the organisation’s 
operation. 

Participants recommended that broad guidelines for writing assessment policy should be 
developed. These could include a list of sub-headings and guidelines for registered training 
organisations regarding the type of information to include when developing policy. 

2. Assessment information provided to learners 

Most registered training organisations represented by workshop participants provide written 
information to students about assessment, but often this is general course information rather than 
relating to assessment specifically. Participants felt that any written information needs to be 
supported by face-to-face or phone contact to provide students with reassurance if necessary. As 
students move through the assessment process, they may require more detailed guidelines. A 
number of participants ensured that both assessors and candidates sign the information as 
confirmation that they had read and understood it. 

Detailed information to candidates is particularly important if the assessment is for recognition 
purposes. If the course is using flexible delivery, training providers need to develop processes to 
ensure that students have access to all relevant information as there is often no formal orientation 
session. 

In some states, the registered training organisation compliance audit requires evidence that 
adequate assessment information has been given to candidates and they have provided proof that 
they understand the procedure. 
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3. Documented oudine of assessment processes 

There was consensus among workshop participants that assessors within registered training 
organisations need to provide evidence of how they go about the assessment process. In some 
organisations, decisions on how to group and assess competencies are made centrally. In other 
cases, these decisions are left to individual assessors. Participants believed that students need to be 
involved in the process of deciding what form of assessment is used. In addition, they felt it 
important that quality assurance involves a feedback loop, thus gathering information from 
candidates about the process. 

Workshop participants were generally aware that their organisation had a documented assessment 
process, but they were not required to keep detailed evidence of candidates having undertaken it 
(with the exception of those participants from Queensland where this documentation was an audit 
requirement). An industry training advisory organisation representative from Queensland — 
currently involved in the state training authority’s audit process — explained that she looks ‘for a 
trail of what has occurred: how did the assessor make the decision that the candidate is competent, 
what evidence did they use and were those pieces of evidence tied together in something that was 
fair, valid, reliable, flexible and consistent?’. 

4. Guidelines for assessment tool development 

In each of the workshops, there was debate about the merit of having standard assessment tools. 
There was agreement that where these tools are used, there needs to be flexibility about how they 
are delivered and assessors should be free to interpret tasks within their own context. 

Participants also felt that if standard assessment tools are used, they need to be continuously 
refined. In some registered training organisations which are delivering statewide training, the 
development of shared tools has advantages in terms of consistency, and they also need to be 
flexible enough to be applied in a range of situations. In addition, participants felt that such tools 
should have benchmarked examples available, preferably with accompanying justification on how 
and why they meet the competency. Participants agreed that the need for benchmarked assessment 
materials was particularly important where training providers are working in partnership 
arrangements with supervisors and managers in workplaces. 

There was agreement that if registered training organisations are developing shared assessment 
tools, they need to provide for industry input into the process. Participants commented that 
sometimes evidence guides in the training package competency standards provide insufficient detail 
and this can lead to different interpretations by assessors. Working with industry can help validate 
the interpretations of the standards made by a training provider. 

5. Processes or materials to support assessors 

Participants felt that support mechanisms for assessors within registered training organisations 
exist, but they are generally of an informal nature. Although there was litde evidence of established 
assessment moderation or verification processes set up within these organisations, there was 
general consensus among workshop participants that such processes can substantially increase 
assessors’ confidence in their decisions. 

Some participants claimed that workplace trainers and assessors are confident in their role as 
industry experts but often need support when taking on new roles as professional trainers and 
assessors. They believed mentoring and team support approaches are essential, particularly where 
the individual is working in isolation in a workplace. Workplace assessors may be uncomfortable in 
their role when it involves assessment of colleagues, and will need support with this changed role. 
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6. Procedures for assessment review 

There was agreement among participants that assessors within a registered training organisation 
need to meet face to face, or communicate by phone or email, on a regular basis to work through 
an assessment validation process. This process requires an agreed structure and may involve some 
form of moderation as well as a review of the assessment processes and procedures. 

Registered training organisation assessment validation practices need to become more widespread 
and should be developed in a format that will suit the groups of assessors and organisations 
concerned. The workshops found little evidence of training providers having documented 
assessment review processes, apart from in Queensland. 

Participants acknowledged the value of technology in supporting assessment review processes but 
felt many assessors were still not comfortable using email or the internet for this purpose. For 
many, sharing assessment material is still a relatively new process. A workshop participant from a 
registered training organisation involved in the delivery of training and assessment in horticulture 
was having considerable success in encouraging teachers to put all their draft teaching programs 
and assessment materials online. He commented that this had initially been a slow process, as 
teachers are often reluctant to distribute their material for peer review. Assessors need to have ‘a 
climate of trust’ to develop confidence to share in this way. 

Focus group members agreed that collecting online resources required a ‘sponsor’ or ‘initiator’. In 
one case, the success of a process for sharing and reviewing assessment resources online was due 
to a manager who encouraged and enthused teachers and collected the ideas together. It was 
particularly relevant for assessors working in information technology to use their technical skills as 
a means of reviewing their assessments. It was also deemed essential for assessments to be 
continually reviewed because rapid technological developments lead to changes in interpretation of 
the standards. 

Some of the larger technical and further education (TAFE) registered training organisations 
conduct a central comparison of certain assessment events that are widely delivered, but this can be 
a time-consuming process. Reference groups, formed as part of the course review process in 
TAFE, can also provide valuable industry input into the validity of assessment tools and processes. 

Participants felt that seeking feedback from students on the assessment process was often 
overlooked by training providers, but it was a necessary part of the review process. 

7. Ongoing professional development for assessors 

Participants agreed that assessors need to maintain currency by conducting assessment on a regular 
basis. Some industry training advisory bodies have interpreted this as doing two assessments in a 
six-month period. One state body proposes keeping a register of current industry assessors. 

There were discrepancies across industries and states about what constitutes industry experience 
and currency. Registered training organisations need guidance from their industry training advisory 
bodies on what experience is required and how they can support assessors in maintaining their 
industry currency. 

Several participants had found taking part in an action-learning process was an ideal way of giving 
assessors an opportunity to interact and develop a shared understanding of the assessment process. 
It also allowed assessors to identify further professional development they may need. 
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Additional feedback from workshop participants 

There was discussion about the meaning of ‘consistency’ and agreement that it should not be 
interpreted as ‘everyone must do the same thing’. Quality assurance depends on the skills and 
experience of all those who are engaged in the interpretation of competence. According to one of 
the private providers working in the community services and health area ‘if it is just the trainer in 
an RTO [registered training organisation] who is making the decision in isolation then there are real 
problems. The process needs to involve students, workplace personnel and supervisors’. 

Participants reiterated the importance of the assessment principles of validity and reliability. They 
felt that assessors must have a shared understanding of the standard and the basis on which 
assessment decisions are made, and what constitutes sufficient evidence. 



Development of recommended strategies to enhance 
assessor confidence 

Feedback from series 1 workshops 

During the series 1 workshops participants worked in facilitated groups, providing feedback on a 
set of draft strategies. Participants evaluated the strategies in terms of how useful they would be to 
their own organisations in increasing assessor confidence. Rather than dismissing the models as not 
useful, participants were asked to comment on how the models could be adapted or elements of 
them incorporated into other strategies. 

Participants also provided valuable information about similar strategies they were aware of, and 
assessors and registered training organisations who should be contacted as possible sources of 
good practice case studies. 



Development of models for series 2 workshops 

The strategies and models listed in table 3 were modified, refined and presented for comment to 
participants in the second series of workshops. Many participants also circulated the models to 
other assessors and managers in their organisation and provided additional feedback to the project 
team. 

Despite the different industry and registered training organisation backgrounds of workshop 
participants, there was general consensus about the degree of usefulness of each of the models 
presented. Participants were also able to identify additional examples from their organisation or to 
nominate training providers they knew had developed models in the areas under discussion. 

The series two models were compiled by the project team based on feedback gathered during the 
first series, and consultation with specific assessors and registered training organisations identified 
as experts in quality assurance assessment issues. Where there was an area of need identified by 
workshop participants and no existing strategies were available, the project team developed new 
draft strategies. One such strategy developed by the project team was an assessment validation 
strategy which all project informants felt was vital for the maintenance of assessor confidence. 

Case study examples of how individual training providers had implemented particular strategies 
were gathered through a series of follow-up face-to-face and telephone interviews with assessors 
and managers. (These included TAPE NSW, Moreton Institute, Regency Institute of TAPE and 
Canberra Institute of Technology.) The case studies were included in the resource to illustrate 
particular strategies and their implementation. 
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Table 4: Participant feedback on models presented during series 1 workshops 



Name and source of model 



Recommendation 



ANTA self-assessment checklist to determine current 
competency (ANTA 1998) 

Assessing competence on and off the job (Kearney 
1997) 

Example of an auspiced partnership agreement, Stanley 
Moriss Assessment and Retail (Enterprise Design 
Associates and TasWRAPS ITAB 1997) 

Moderation (Group) (Foyster 1995) 

Internal moderation arrangements (London Open 
College Network 2000) 

School of Computing and Information Systems Quality 
Procedures. Internal moderation. (School of Computing, 

Engineering and Technology, University of Sunderland 
1997) 

In moderation — assessment review (New Zealand 
Qualifications Authority 1999a) 

Sampling (Konrad 1999) 

Managing the validity and reliability of assessment 
(Bateman unpublished) 

TAPE Tasmania — Workplace Learning Services 
(Material sourced from Enterprise Design Associates 
and Michelle Harwood, Trans Train, unpublished) 

Notes: NVQ = national vocational qualifications 

RTO = registered training organisation 

The case studies were structured using a series of suggested sub-headings provided by feedback 
from assessors in the workshops. The set of questions used in interviews with training providers 
were: 

What is the context of the strategy that your RTO [registered training organisation] has 
developed? (What training package, delivery scope etc.) 

What is the approach that your RTO has taken? What area in the assessment process does it 
focus on? 

What is the value of this approach? 

Summary of participant feedback on models presented during 
series 2 workshops 

Series 2 workshop participants were asked to provide feedback on the usefulness and possible 
modification of a series of eight quality assurance models. These were provided by participants 
during the first series of workshops or developed by the project team following participant 
feedback in the earlier workshops. 

Participants were provided with background information about each of the models and then 
worked in groups analysing the strengths of the models. They were encouraged to suggest possible 
modifications or identify more appropriate examples rather than rejecting the models completely. 



Useful. Many assessors may not be aware of it. Include 
in final product. 

Useful description of moderation process. Elements to 
be included in guidelines for assessment validation. 

Thorough, detailed example of an agreement. Useful to 
include in final product. 

Incorporate relevant sections into guidelines for RTOs on 
establishing an internal assessment validation strategy. 

Useful if rewritten for the Australian context. Re-draft. 

Inappropriate for Australian VET context. Some sections 
to be modified and included in an assessment validation 
strategy. 

Not useful for Australian context. Locate examples of 
RTOs assessment review plan. 

Of interest to larger RTOs. Re-work and include with 
validation strategies. 

Include an assessment principles checklist in the final 
product. Other potential sources identified. 

Useful. Redraft, and develop supporting documents. 
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Table 5: Summary of feedback on models from series 2 workshop participants 



Name/source of model Description 



Feedback/recommendation 



Diagnostic assessment 
toot 

(Extract from Thomson, 
Saunders & Foyster 2001) 



Diagnostic tool to improve the 
quality of assessment. Consists of 
short questionnaire with follow-up 
material. 



Good for reflective thinking and making the 
assessors think about the tool. 

Time-consuming for everyday use. Format and 
bulk of this model may not be appropriate. 
Include in draft kit for further feedback. 



Witness testimony form 

Key principles for using 
witness testimony as 
performance evidence 
(Centre Undertaking 
Research in Vocational 
Education and Training, 
unpublished) 

Assessment principles 
checklist 

(Vocational Education and 
Assessment Centre 2001) 



Pre-assessment 
verification of 
assessment task 

(Document developed by 
project team, unpublished) 

Establishing internal 
validation processes 

(Document developed by 
project team, unpublished) 



Industry validation of 
assessment 

(Draft document developed 
by Department of 
Horticulture, Canberra 
Institute of Technology, 
unpublished) 

Guidelines for the 
development of 
assessment instruments 

(Developed from material 
supplied by TAPE SA, 
unpublished) 

Building confidence in 
witness testimony or 
third-party evidence 

(Document developed by 
project team, unpublished) 



To be used by an assessor in an 
RTO when collecting third-party 
evidence about a candidate. 
Includes a form to ensure that the 
witness is aware of all facets of the 
assessment and collect the 
evidence. 



Checklist for assessors organised 
under the assessment principles: 
validity, reliability, fairness and 
accessibility, flexibility and cost- 
effectiveness. 



To be used as part of an 
established assessment validation 
process. 



Draft material for assessors and 
RTOs about the establishment of 
internal validation processes to help 
maintain the quality of assessment. 



Draft case study Department of 
Horticulture, Canberra Institute of 
Technology. Involved industry in 
development of assessment 
methodology to validate 
assessment processes and 
documentation. 

Material provided by TAPE SA 
during the first stage of project has 
been used for these guidelines for 
assessment tool development. 



Guidelines for assessors in the use 
of witness testimony or third-party 
evidence as part of the assessment 
process. 



Potential to create better partnerships when 
more than one party is involved. Very useful 
for Australian Qualifications Framework 1 , 2 
and 3. Form needs to be re-worked with space 
for qualitative comments. Rename as third- 
party evidence form. Edit form and include in 
draft kit for further feedback. 



Useful for ongoing assessor professional 
development or as a self-access checklist 
used to evaluate assessment design. Useful 
tool for discussion with staff, for keeping 
records and accountability or as part of an 
assessment validation process. 

Simple, effective and easy to use. A tool that 
does verify and give confidence if used 
consistently. Useful for self-assessment, with 
a peer or mentor or as part of a verification 
strategy. 

Useful model. Needs someone to take up the 
role of a co-ordinator. Could scale down the 
process for smaller groups/RTOs. Helps 
promote networking 

User-friendly, good working document. 
Validation action plan needs to be re- 
designed. 

Idea of a case study and flow chart very 
useful. Modify and include in draft kit. Need to 
source other case studies to illustrate a 
number of the other strategies included in the 
kit. 



Useful for awareness-raising, refreshing 
memory, checking appropriate use of 
instruments. Can help to encourage use of a 
greater variety of tasks. There are many 
versions of this type of resource around. 
Needs more information about combining 
assessment activities. Do not include in kit. 

Useful and succinct. Clarifies the importance 
of the process. Needs more guidance about 
when to use witness testimony. Modify and 
add to draft kit. 



Note: RTO = registered training organisation 



Recommended resource format 

Feedback was provided about the format of the proposed resource. Participants felt the strategies 
provided would work very well in CD ROM format or on a website, so that registered training 
organisations and assessors could use and customise as needed. 

Other recommendations included the need for a table of contents, and a short introduction stating 
how the resource should be used. Participants also believed it was important to have an 
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understanding of the intended audience, and stressed the value of implementing quality assurance 
strategies for assessment. They felt there should be a glossary of terms included with the resource 
and that some additional case studies to support the models would be useful, particularly in the 
area of assessment verification. 

Development and review of draft resource 

The project team drafted a kit of strategies and models to assist registered training organisations 
and assessors deal with quality issues relating to assessment. This was based on data collected on 
current strategies and the feedback gathered on proposed models, templates and case studies 
through workshops and consultation. 

The kit of draft resources was circulated for comment to a group of reviewers from a range of 
selected training providers and diverse industry backgrounds. The group included five 
representatives from industry areas that had not been represented in the project workshops. 
Reviewers were asked to complete a feedback sheet to enable easy collation of comments and a 
number of reviewers were interviewed by phone to collect their feedback. 

Reviewers were asked to provide general feedback on the appropriateness of the overall approach 
used in the kit, and how applicable the resource would be for use by the registered training 
organisation. Reviewers were also asked to rate how ‘user friendly’ they found the resource and 
how the language and design of the document could be improved. Detailed feedback on the 
content of each section of the kit as well as recommendations for improvement were also sought 
from reviewers. 

The final version of the kit has been organised into three sections, which were considered by those 
consulted during the research as the most helpful in improving and maintaining assessor quakty. 
These sections are assessment review strategies, gathering evidence, and partnerships and 
networks. Strategies refined through the research project, short case studies, and templates to 
support registered training organisations in the implementation of particular strategies have been 
grouped into these sections. The document has been designed as a resource to cater for the diverse 
range of training providers currendy assessing and awarding a qualification as part of a training 
package; thus, not all strategies will be relevant to every organisation. The range of strategies 
included is a reflection of the wide variety of participants involved in the research project. 
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Appendix 1 



Maximising confidence in assessment decision-making; 

The issues 

This paper has been prepared as part of the first stage of a National Research and Evaluation 
Committee project entitled Maximising Confidence in Assessment Decision-making: Current 
Approaches and Future Strategies for Quality Assurance. The contents are designed to provide a 
basis for discussion by participants involved in the workshops in the next stages of the project. 



Brief background to the project 

The National Product/Service Standard for Training Delivery TD3 requires registered training 
organisations to demonstrate that they conduct assessments in accordance with the endorsed 
components of the training packages they are delivering. 

The National Assessment Principles emphasise that assessment is to be conducted within a quality 
assurance framework. Thus, registered training organisations are required to monitor and evaluate 
assessment practices within their organisation and within any auspicing arrangements that may be 
established between enterprises and the organisation. 

This emphasis on quality assurance is designed to develop and maintain a sense of confidence in 
assessment decision-making, because mutual recognition and qualifications are dependent on ready 
acceptance of the outcomes of the assessment process. Additionally, the introduction of training 
packages is requiring assessors to work in different ways, in new locations and possibly in 
partnership with others. Such changes require assessment to be monitored and evaluated to ensure 
that it remains valid, reliable, fair and credible. 

Research questions and outcomes 

Some of the research questions that are being addressed by this project are: 

“k What factors contribute to the development of confidence in assessment decision-making? 

What strategies are assessors using to ensure that their judgements of competence are valid, 
reliable, fair and based on the collection of sufficient evidence? 

“k What strategies are registered training organisations using to support the verification and 
validation of assessment decisions, and how cost-effective are they? 

"k What quality assurance mechanisms do assessors, managers and industry training advisory 
bodies consider essential, particularly with the implementation of training packages? 

The objectives of the research are to provide resources to support confident decision-making by 
assessors in vocational education and training, as well as guidelines and scenarios for the 
establishment of cost-effective verification and validation strategies for use by assessors. In 
addition, it aims to formulate strategies for registered training organisations to assure the quality of 
assessment decisions made by their assessors. 
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In parallel with this project, two other projects are looking at the technical aspects of validity and 
consistency in assessment. Researchers working on all projects will maintain contact to ensure that 
the best possible outcomes are achieved. 

Definitions 

In the initial stage of this research, the definitions for quality assurance, verification process and 
validation process provided in the Training Package for Assessment and Workplace Training 
(ANTA 1998) will be used. They are: 

Qualily assurance: A planned and systematic process of ensuring that the requirements of the assessment systems, 
competency standards and any other criteria are applied in a consistent manner. Quality assurance mechanisms or 
procedures are an integral part of an assessment system. 

d}- Aerification process: Is the means of ensuring that the assessment decision is consistent and reliable. The process 
may involve having another assessor(s) confirm the assessment decisions. 

A Validation process: process to ensure assessment tools, procedures and decision criteria lead to a correct decision 
when used to assess competency. The process may involve having both technical and assessment specialists review 
the assessment tools and procedures for validity. 

While there is no absolute definition of the term ‘confidence’, for this project it represents a sense 
of trust in the quality of the information provided to learners, the validity of assessment tools, and 
the reliability and fairness of assessment procedures. It also involves the sufficiency of evidence, 
correctness of the interpretation, and the accuracy of the recording and reporting of assessment 
results. 



The findings so far 

The first stage of this project has entailed: 

A a literature review to determine the extent of research undertaken in the area of quality 
assurance of assessment 

A a focus group of practitioners and managers from a range of registered training organisations 
with representatives from industry training advisory bodies and group training companies. The 
role of the participants in this focus group was to find out the issues and concerns associated 
with making assessment decisions, particularly in the changing environment generated by the 
introduction of training packages. 



What the literature says about approaches to quality assurance in assessment 

While there is considerable literature on validity, reliability and moderation in educational testing, 
particularly in the United States, there is limited material on strategies for quality assuring 
assessment in a competency-based environment. 

International approaches: Central regulation 

Some of the most pertinent literature on quality assurance in assessment addresses the approaches 
adopted in the delivery of vocational education and training in the United Kingdom and New 
Zealand. Both countries use a centrally directed, regulatory methodology with extensive forms of 
moderation to ensure quality outcomes. 

In the United Kingdom, for example, the Awarding Bodies’ Common Accord 1997 provides the 
guidelines for quality assurance arrangements in national vocational qualifications. In addition. 
Implementing the National Standards for Assessment and Verification sets out how the national 
standards units are to be assessed and verified. 
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As in Australia, training providers are required to undergo registration and, as part of this process, 
they must demonstrate their capacity to undertake assessment. In addition, they must implement 
internal moderation procedures and external verifiers must determine whether the quality of 
assessment and verification meets national standards. The strategies used to ensure that assessment 
decisions are valid, fair and consistent across training providers are: 

visitation moderation 

sampling of candidate assessments 

monitoring and evaluation of assessment and verification practice. 

The United Kingdom system provides a range of possibly useful strategies for verification and 
validation of assessment in the Australian VET sector. Some of these will be used as models for 
examination in this stage of this project. Of particular interest are: 

Internal verification ofN]/Qs:Amide for internal verifiers (National Council for Vocational 
Qualifications 1997b) 

internal moderation arrangements (London Open College Network 2000) 

internal moderation arrangements (School of Computing, Engineering and Technology, 
University of Sunderland 1997) 

guidelines for witness testimony and simulations (Scottish Qualifications Authority 1 997) 
support for assessors (National Council for Vocational Qualifications 1997b) 
quality standards (Konrad 1999). 

In examining these models, it is also important to consider research undertaken on the 
effectiveness of the national vocational qualification/ general national vocational qualification 
quality assurance system in the United Kingdom. 

In research undertaken on the delivery of the national vocational qualifications for the agricultural 
sector, Lester (1999) found that both employers and providers are concerned with assessment 
decision-making and, in particular, with decisions about competence. Additionally, there appears to 
be very limited confidence in the quality assurance system. The lack of knowledge and consistency 
on the part of external verifiers is seen as a particular problem. 

In commenting on the review of the United Kingdom system, Lester (1996) emphasises the link 
between the credibility of national vocational qualifications and quality assessment. He questions 
whether increasing the level of monitoring, and the use of external verification and standardisation, 
are likely to ensure quality. He concludes that while there may be a greater sense of public 
confidence generated by these processes of internal and external verification, validity may be 
adversely affected because assessment increasingly concentrates on elements which are amenable to 
checks and controls. 

In contrast to the quality control approach, there appears to be considerable support for placing 
the emphasis on quality assurance and, in particular, focussing the effort on developing the skills 
and knowledge of assessors (Black 1993; Eraut 1994; Lester 1996). Eraut contends that 
inconsistencies in assessment decision-making occur despite the fact that detailed criteria are 
available, simply because the initial training and ongoing support for assessors is neglected. The 
importance of communication including open and regular discussion and networking are seen to be 
critical to the development of a ‘community of assessors’ (Eraut 1994). 

Furthermore, there are particular advantages in assessors sharing their assessment and verification 
experiences. As Black notes: 

... if they can focus on problems of interpretation, perhaps through sharing real examples of 
work or descriptions of contexts, the difficulties of translating written standards into useable 
criteria can be overcome and a local, if not national comparability will emerge. 

(Black 1993, p.5). 
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In implementing the National Qualifications Framework, the New Zealand Qualifications 
Authority adopts a somewhat less rigorous approach. It requires training providers to develop their 
own quality assurance systems. However, all unit standards set requirements for moderation, and 
training providers and standards-setting bodies are responsible for ensuring that assessors 
participate in the designated process for ensuring validity and consistency in assessment. Quality 
assurance strategies include: 

moderation action plans 

consensus moderation to determine comparability of assessment decisions across a range of 
training organisations 

national networking of subject moderators 

evaluation of assessment systems at the time of accreditation/ re-accreditation. 

The New Zealand Qualifications Authority has recently piloted a new assessment review process in 
schools and this is included as a model for examination in this project. In this example, an 
assessment review plan is developed for each school, which identifies the subjects to be moderated 
and the number of assessment tasks and products that will be examined on an annual basis (New 
Zealand Qualifications Authority 1999b). 

Again, the strategies employed in the New Zealand system may provide some insights or guidance 
on formulating quality assurance approaches in VET sector assessment in Australia. 

The A-Ustralian approach: A^n emphasis on polig and self-regulation 

Given the deregulated nature of the Australian training system, the concept of a centrally driven 
regulatory approach — such as that adopted in the United Kingdom or New Zealand — appears to 
have little application. In the Australian setting, self-regulation is the clear focus of policy 
underpinning VET delivery and assessment. It would seem more feasible, therefore, to formulate 
strategies which are capable of being implemented and managed from the ‘bottom-up’, rather than 
centrally imposed from above. 

Despite the existence of a strong policy framework to support quality VET outcomes, however, 
the literature on competency-based training in the VET sector in recent times has contained 
considerable comment about the need for quality assurance in assessment (Docking 1998; Gdlis, 
Griffin, Trembath & Ling 1998; GiUis & Bateman 1999; Bateman unpublished). Much of the 
discussion has centred on concerns about consistency and the need for registered training 
organisations to incorporate ongoing processes of monitoring and review. This emphasises the 
obligation that training providers have to implement such procedures under the requirements of 
the Australian Recognition Framework and the associated National Assessment Principles (ANTA 
1998). 

GiUis, Griffin, Trembath and Ling (1998) suggest that at the system level, quaUty assurance 
mechanisms can and should be buUt into the VET system in a number of ways. In particular, they 
identify the foUowing as critical quaUty assurance mechanisms: the assessment guidelines and 
additional assessment resources contained in the nationally endorsed training packages, and the 
monitoring and auditing of training organisations through the registration process. 

The same authors also emphasise the need for quality assurance mechanisms at both the 
organisational and assessment level. At the organisational level, the major components are 
monitoring and auditing of assessment together with quality training and development of assessors. 
WhUe at the assessment level, they reinforce the need for moderation and verification of decisions 
and procedures, mechanisms for appeals and comprehensive record-keeping systems. 

Foyster (1995) defines moderation as ‘a fault-correction device designed to achieve improvement at 
the lowest possible cost’ (p.l35). Moderation involves systematically sampling assessment products 
and outcomes to determine the validity and reUabUity of the decisions that are made. Foyster 
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provides an appraisal of the applicability of statistical, visitation and group moderation to a 
competency-based setting and concludes that each requires a substantial financial commitment. 

He also suggests that group moderation, whilst costly, has some benefits in that it encourages the 
maintenance of standards and the establishment of formal and informal networks which are 
beneficial to ongoing assessor development. 

Bateman (unpublished) stresses the importance of registered training organisations developing 
appropriate assessment documentation and processes. Formulating policy and strategies to help 
develop assessment processes, guidelines and instruments will improve the validity and reliability of 
the assessment outcomes. 

Both the documentation and processes constitute an assessment system that requires ongoing 
evaluation and maintenance. Bateman suggests the continuous improvement cycle for an effective 
assessment system involves: 

quality assuring the assessor process and the assessor 

quality assuring the assessment judgements 

quality assuring the procedures 

professional development of assessors. 

The Email (1998) project Best Practice Assessment Systems and Processes also identifies the need 
for senior management support, quality documentation, suitably qualified assessors, and an 
assessment system that is continuously reviewed and improved. 

In a recent review of the literature on competency-based assessment, Gdlis and Bateman (1999) 
nominate strategies for improving the validity and reliability of assessment in the VET sector. The 
suggestions have particular relevance for individual assessors. They also need to be recognised as 
important by managers working to set up quality assessment processes within registered training 
organisations. The requirements are extensive, and decisions about quality versus cost are likely to 
generate discussion prior to implementation of a number of the activities proposed. Gdlis and 
Bateman state: 

Ultimately, the validation of assessment in terms of reliability and validity requires evidence of 
careful task development, clear and concise assessment criteria against the competency 
standards, appropriate task administration procedures, and adequate scoring/ decision-making 
rules and recording procedures. (GiUis & Bateman 1999, p.32) 

Cost wid be a significant factor in the acceptance and implementation of any quality assurance 
framework by a training provider. The issue of costs of assessment in training packages is the focus 
of a report undertaken for the Western Australian Department of Training by Docking (1998). He 
offers some suggestions for improving the quality of assessment whde minimising the cost of the 
process. These include providing training and exemplars on the internet to assist in the 
development and interpretation of benchmarks for assessment, and the development of assessment 
tools which can be accessed through an internet-based resource library. 

Tower, Bloch and Harvey (1999) suggest that technology such as teleconferences and training 
provider intranet sites can be used to improve communication between assessors and generate 
debate about the issues, thus raising the skdls and understanding of assessment in the VET sector. 

Assessor training programs and consistency and assessor skdls have been dnked in several reports 
(Gdds, Griffin, Trembath & Ling 1998; Gdds, Griffin, Catts & Falk 1998). The quadty of the 
assessment process and judgements is very much dependent on the quality of the training that 
assessors are able to access. In their investigation of summative assessment practices in TAFE 
NSW, Tower, Bloch and Harvey reinforce this view and note: 

. . . clearly there are no simple answers to improving consistency, however, we need to bear in 
mind the broader question of how much of the assurance of quality in assessment can be 
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based on regulation and standardised procedures, as opposed to developing teachers as 
skilled, confident assessors. (Tower, Bloch & Harvey 1999, p.2) 

The review of the literature reveals that Australian information on possible models and working 
examples is relatively limited. However, the following have been drawn from the literature for 
examination in this project: 

self-assessment checklist to determine current competencies (ANTA 1998) 

self-assessment checklist (Bateman unpublished) 

strategies to minimise cost and maximise quality (Docking 1998) 

group moderation (Foyster 1995) 

validity and reliability checklists (Bateman unpublished). 

Each example has the potential to be used by individual or groups of assessors within a registered 
training organisation to support improved assessment practice and decision-making. 



Outcomes from the focus group 

A focus group was conducted in Sydney on 16 November. The 25 participants were currently 
involved in competency-based assessment or in the quality assurance of VET outcomes. 

Each participant was provided with a set of questions prior to the day which were designed to 
provide a framework for the discussions. The questions were: 

What do you think constitutes quality assessment? 

What strategies have been used in recent years to assure the quality of assessment decision- 
making in your own training environment? 

With the implementation of training packages, how is assessment different? 

What are the critical issues/ aspects relating to assessment decision-making faced by individual 
teachers/ trainers/ assessors and RTOs [registered training organisations] 

The questions generated considerable debate, and participants raised a number of issues perceived 
to be having an impact on the quality of assessment and the level of confidence of assessors and 
others in the VET sector. These issues fall under the following headings: 

1 . The lack of consistency in assessment decisions and assessment practice. 

2. The new demands placed on assessors and assessment with the implementation of training 
packages. 

3. The lack of rigorous quality assurance processes. 

4. Concerns about the quality of assessor training programs and ongoing support for assessors. 
Each of these is now discussed in greater detail. 

1. The lack of consistent in assessment and assessor practice 

Participants talked about the inconsistency of approaches to assessment in the VET sector, despite 
the implementation of national curriculum and competency standards, training packages with 
assessment guidelines and supporting assessment materials. 

They particularly noted that the ‘language’ of assessment is confusing, leading to an uneven 
understanding and application of assessment principles and guidelines. 

The problem of inconsistency particularly affected the area of mutual recognition, with fears that 
this would lead to mistrust and a lack of confidence in the assessment decisions being made. 
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Participants consider inconsistency arises when assessors: 

are unable to interpret the requirements of competency standards 
lack currency in the technical domain 

do not review their assessment processes to determine irregularities 

do not have the skills to generate assessment tasks which draw on a sufficient range of evidence 
do not share resources, information and discussions about assessment. 

Some good examples of strategies to minimise inconsistency are: 
developing common assessment tasks 
using assessment panels 

providing assessors with assessment kits which guide the process 
having access to quality assessment tools 

networking to discuss assessment issues and specific assessment tasks. 

The sharing of assessment resources and information is also seen as one way of making assessment 
more consistent. However, some participants felt that this is problematic in a competitive training 
environment. 

2. The new demands placed on assessors and assessment with the implementation of 
training packages 

Under this heading, issues identified related to specific concerns about assessment in training 
packages, and trust or confidence in the assessors themselves and other partners in the assessment 
process. 

In relation to training packages, concerns focussed on: 

how and what underpinning knowledge is to be assessed 

interpretation of the standards to ensure benchmarks are achieved 

k- determining competence in an off-the-job training environment 

given the stress on work-based assessment, the role and extent of simulation as a method of 
assessment 

accessing relevant workplaces and appropriate work tasks for assessment of learners completing 
training in off-the-job environments 

provider and industry roles and responsibilities in the assessment and verification of assessment 
decisions. 

Issues of trust and confidence focussed on: 

the authenticity and ready acceptance of witness testimony/ third-party evidence 

k- registered training organisation and industry partnerships in assessment 

the ability and assessment decisions of other assessors 

k- the effectiveness of auspicing arrangements 

personal ability to take on assessment in a range of new locations and contexts 

k- the value which employers, industry and other users give to assessment that is undertaken off 
the job. 

Ongoing professional development — which involves working with training packages, networking 
with other assessors and gaining access to quality assessment tools — is seen as a means of solving 
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some of these issues. Examples of the experiences of others undertaking assessment in partnership 
in workplaces will also assist in clarifying some of the problems and how to address them. 

3. The lack of rigorous quality assurance processes 

Participants acknowledged that various national policies provide direction on the monitoring and 
review of competency-based assessment in the VET sector. However, they agreed that there is a 
need for concrete procedures to be put in place by training providers. Some people, however, are 
concerned about the degree of organisational support likely to be available for quality assessment 
processes. 

The major issues are seen to be: 

the cost of setting up and maintaining such strategies 

ri the appropriate identification of roles and responsibilities for quality assurance when assessment 
is undertaken in different locations, possibly in a collaborative way 

ri the limited understanding and skills that people have of moderation and sampling processes 

ri the need for training to support auditing assessment processes and decisions 

ri that it is not a job for either individuals or the untrained 

ri that it would put considerable stress on already over-worked people within registered training 
organisations 

ri that auditing would have to be performed by external auditors. 

However, it is generally agreed that assessment should be a focus when training organisations are 
audited, especially when strategies for ongoing improvement are an essential component of the 
national quality system. 

Some of the strategies suggested are: 

developing a simple system within registered training organisations where assessment tools are 
submitted to someone for scrutiny or review 

following up on examples of poor assessment practice 

making assessment a focus of the continuous improvement cycle that training providers are 
implementing 

ri ensuring that when reviews of assessment are completed, feedback is provided. 

4. Concerns about the quality of assessor training programs and ongoing 
support for assessors 

Assessor training and the ongoing maintenance and development of assessors’ skills were seen as 
the critical element of any quality assurance system. Selection, initial training and appropriate and 
ongoing assessor development were covered in the discussions. 

A number of participants in the focus group raised concerns about the initial selection and training 
of assessors. They suggested that assessors should not only be carefully chosen, but also needed to 
be genuinely interested in undertaking the job. 

Assessor training programs are seen as a matter for concern. Examples were given of training 
programs that are quite short, while others are considerably longer and appear more 
comprehensive. Participants see such diversity in assessor training as likely to influence the degree 
of confidence that others have in the quality of the assessment decisions that are made by some 
assessors. They are also concerned about the equity and industrial relations issues that such 
variations in assessor qualification may bring. 
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It is generally agreed that, to be current, assessors need to conduct assessments regularly to ensure 
that their skills and knowledge are used and constantly developed. If assessors are not assessing 
regularly, they may need to re-do their training. 

At the same time, there was considerable discussion about how difficult it was becoming for off- 
the-job assessors to access time back in relevant workplaces to maintain their technical skills and 
knowledge. This is seen as particularly true in TAPE organisations, where there is limited money 
available for return-to-industry. In contrast, sessional and contract staff are seen as having current 
industry skills, but less opportunity to build their assessment skills and knowledge through ongoing 
professional development. 

It was noted that the neglect of teacher training and casualisation will undermine all strategies for 
quality which are put in place. 

Focus group participants agreed that a lack of currency in either the technical or assessment 
domain generates a loss of confidence in the assessment process, and affects employers, industry 
and other users of the results. The assessors themselves also feel less confident in their abilities. 

Workshops, seminars, networking, online help, and re-training can all help assessors. The concept 
of teaming assessors lacking current technical skills with assessors who are up to date or mentoring 
assessors was also suggested. These are possible ways of maintaining and enhancing assessor skills, 
but the cost of such strategies is acknowledged as a problem. 



Summary 

The Australian Recognition Framework and the quality arrangements in place for all registered 
training organisations provide the essential framework for ensuring the quality of assessment in the 
VET sector. Participants in the focus group in the first stage of this project, however, have 
highlighted a range of issues they see as impacting negatively on assessment and confidence in 
assessment outcomes. Significant issues include the uncertainty generated by inconsistencies in 
assessment practice; the new requirements of training packages; the lack of rigorous quality 
assurance of assessment within training providers, and concerns about initial and ongoing training 
for assessors. 

The literature on moderation and verification processes in the United Kingdom and New Zealand 
provides a number of models worthy of closer examination, as do some recent research and 
development activities from within the VET sector itself. The literature also raises some of the 
same concerns about assessment as those identified by practitioners in the focus group. Questions 
were asked about how well assessors are being prepared to carry out their role and whether they 
have the skills and knowledge to develop valid assessment instruments, interpret standards and 
determine competence. At the same time, the literature provides clear warnings about taking a 
quality assurance, rather than a quality control, approach when developing strategies to maximise 
confidence in assessment decision-making. This too requires closer investigation in this project. 
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Appendix 2 



Workshop and focus group participants 



Name 

Pat Alexander 
Susie Allen 
Janice Anderson 
Kirsten Bailey 
Sue Blyth 
John Brereton 
Margaret Broun 
Jan Brown 
J ohn Brunskill 
Melissa Cerantola 
Carol Christie 
Marion Clegg 
Debbie Cole 
Leone Cripps 
Jo Crothers 
Tony Dodson 
Gail Evans 
Gill Fergie 
Warren Finch 
Ken Foote 
Clare Forbes 
David Frith 
Shirley Gerrard 
Justine Gerr)^ 
Dianne Hardman 
Shirley Harring 
Marie Healy 

Carolyn Hildebrand 
Anne Houghton 

Maureen Imeson 
Bernadette loannou 
Judy Johnston 
Anna Johnston 
Lyn J ordan 
Maureen Joyce 



Organisation 

Torrens Valley Institute of TAFE 
TAFE Tasmania 
Canberra Institute of Technology 
TAFE Tasmania 
TAFE NSW 

University of Melbourne, Burnley 
NSW Department Education & Training 
Hawkesbury Family Day Care 

Northern Group Training Company 
Moreton Institute of TAFE 
Moreton Institute of TAFE 
Swinburne University of Technology 
Network of Community Activities 
TAFE Tasmania 
HORTIS, South Australia 
G E Consultancy Pty Ltd 
Office of Training and Adult Education 
TAFE NSW 

Hunter Valley Group Training Company 
Swinburne University of Technology 
TAFE NSW 

Dickson College, Australian Capital Territory 
Customs Brokers Council of Australia 
TAFE Tasmania 

Australian Child Care Career Options 

Open Learning Institute, DETIR (Department of Employment, 
Training and Industrial Relations, Queensland) 

Brisbane Institute of TAFE 

Office of Training and Adult Education, Australian Capital 

Territory 

Box Hill TAFE 

TAFE NSW 

Northern Melbourne Institute of TAFE 

Lady Gowrie Child Centre 

Northern Melbourne Institute of TAFE 

OTEN-DE (Open Training Education Network-Distance 

Education) 



NCVER 
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Name 

Robyn Knox 
Ros Lamprill 
Sandra Lawrence 
Peter Le Cornu 
Charles Lenard 
Glenyss Leyne 
Helen Lumby 
Jan Macindoe 
Kathie Mackay 
Judy Maggiolo 
Debbie May 

Louise Mayo 
Margaret McCullough 
Julie McQueen 

Robyn Monro Miller 
Sharon Mills 
Maria Minasi 
Joy Mitchell 
Ros Morgan 
Jan New 
Sonya Oper 
Elizabeth Owers 
Louise Strode Penny 
Barr^r Porter 
Amanda Porter 
Claire Ralfs 
Graham Ratcliff 
Natalie Reed 
Rosie Ryan 
Michael Scutter 
Jenny Seymour 
Sue Shrubb 
Diana Smith 
Brian Spencer 

Judy Swift 
Brian Thomas 
Judy Thompson 
Belinda Tierney 
Pat Treacy 
Leo Van Neuren 
Jennie Wallace 
Tracey Worrall 

Liz Wright 



Organisation 

The Australian College of Applied Science 
TAPE Tasmania Clarence Campus 
Brisbane Institute of TAPE 
Canberra Institute of Technology 
TAPE NSW 

Rural Training Council of Australia 
Office of Training and Adult Education, ACT 
TAPE NSW 
TAPE NSW 

Brisbane Institute of TAPE 

NSW WRAPS (Wholesale Retail and Personal Services Industrj^ 
Training Council) 

Australian Business Academy 
Department of Education and Training 

MINTRAC (National Meat Industr)^ Training Advisor}^ Council 
Ltd) 

Network of Community Activities 

Darebin Childrens Services (Local Government) 

Russo Institute 

Manufacturing Learning Victoria 

Chisholm Institute, Victoria 

Sutherland Shire Council 

Continuing Education 

Regency Institute of TAPE, South Australia 

Open Learning Institute of TAPE Queensland 

TAPE NSW 

Sigma Pharmaceuticals 

COPE (Centre of Personal Education), Adelaide 

TAPE Tasmania 

Pamily Day Care 

Chisholm Institute of TAPE 

Adelaide Institute of TAPE 

Canberra Institute of Technology 

Northern Sydney Institute of TAPE 

Southern Sydney Institute of TAPE 

Community Services and Health Industr}^ Training Board 

Victoria 

Bendigo Regional Institute of TAPE 
TAPE NSW 

Moreton Institute of TAPE 
Northern Group Training Company 
Lake Ginninderra College, ACT 
Communications Industr)^ Training Advisory Board 
Study Group Australia 

Queensland Community Services and Health Industry Training 
Council 

Community Services and Health Industty Training Board, 
Victoria 



42 



Maximising confidence in assessment decision-making 





NCVER 



The National Centre forVocational 
Education Research is Australia’s 
primary research and development 
organisation In the field of vocational 
education and training. 

NCVER undertakes and manages 
research programs and monitors the 
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system. 
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